Optical system using segmented phase profile liquid crystal lenses

ABSTRACT

A system is provided. The system includes a display configured to output a virtual image. The system also includes a lens assembly optically coupled to the display and including a plurality of optical lenses. The system also includes a controller configured to selectively activate one or more of the plurality of optical lenses, determine a lens center shift between a center of the selectively activated one or more of the plurality of optical lenses and a center of the lens assembly, and determine an image shift based on the lens center shift for shifting the virtual image output from the display.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/261,552, entitled “OPTICAL SYSTEM USING SEGMENTED PHASE PROFILELIQUID CRYSTAL LENSES,” filed on Jan. 29, 2019, which claims the benefitof priority to U.S. Provisional Patent Application No. 62/780,202, filedon Dec. 15, 2018. Contents of the above-mentioned applications areincorporated herein by reference in their entirety.

BACKGROUND

Virtual reality (VR) headsets can be used to simulate virtualenvironments. For example, stereoscopic images can be displayed on anelectronic display inside a headset to simulate the illusion of depth,and head tracking sensors can be used to estimate what portion of thevirtual environment is being viewed by the user. However, becauseexisting headsets are often unable to correctly render or otherwisecompensate for vergence and accommodation conflicts, such simulation cancause visual fatigue and nausea of the users.

Augmented Reality (AR) headsets display a virtual image overlapping withreal-world images. To create comfortable viewing experience, the virtualimage generated by the AR headsets needs to be displayed at the rightdistance for the eye accommodations of the real-world images in realtime during the viewing process.

BRIEF SUMMARY OF THE DISCLOSURE

One aspect of the present disclosure provides a system. The systemincludes a display configured to output a virtual image. The system alsoincludes a lens assembly optically coupled to the display and includinga plurality of optical lenses. The system also includes a controllerconfigured to selectively activate one or more of the plurality ofoptical lenses, determine a lens center shift between a center of theselectively activated one or more of the plurality of optical lenses anda center of the lens assembly, and determine an image shift based on thelens center shift for shifting the virtual image output from thedisplay.

Other aspects of the present disclosure can be understood by thoseskilled in the art in light of the description, the claims, and thedrawings of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a relationship between vergence and eye focal lengthin the real world;

FIG. 1B illustrates a conflict between vergence and eye focal length ina three-dimensional (3D) display screen;

FIG. 2A illustrates a wire diagram of an example head-mounted display(HMD) according to the present disclosure;

FIG. 2B illustrates a cross-section of a front rigid body of the HMD inFIG. 2A according to the present disclosure;

FIG. 3 illustrates an exemplary varifocal structure according to thepresent disclosure;

FIGS. 4A-4E illustrate an exemplary segmented phase profile (SPP) LClens according to the present disclosure;

FIG. 5 illustrates an exemplary configuration of two stacked SPP LClenses according to the present disclosure;

FIGS. 6A-6C illustrate exemplary configurations of an SPP LC lens arrayin a single lens layer according to the present disclosure;

FIG. 6D illustrates exemplary lens center shift and image shiftaccording to the present disclosure;

FIG. 7 illustrates an exemplary varifocal system in which an HMDoperates according to the present disclosure; and

FIG. 8 illustrates an exemplary adaptive lens assembly adjusting processaccording to the present disclosure.

DETAILED DESCRIPTION

Vergence-accommodation conflict is a problem in many virtual realitysystems. Vergence is the simultaneous movement or rotation of both eyesin opposite directions to obtain or maintain single binocular vision andis connected to accommodation of the eye. Under normal conditions, whenhuman eyes look at a new object at a distance different from an objectthey had been looking at, the eyes automatically change focus (bychanging their shape) to provide accommodation at the new distance orvergence distance of the new object.

FIG. 1A shows an example of how the human eye experiences vergence andaccommodation in the real world. As shown in FIG. 1A, the user islooking at a real object 100 (i.e., the user's eyes are verged on thereal object 100 and gaze lines from the user's eyes intersect at realobject 100). As the real object 100 is moved closer to the user, asindicated by the arrow in FIG. 1A, each eye 102 rotates inward (i.e.,convergence) to stay verged on the real object 100. As the real object100 gets closer, the eye 102 must “accommodate” for the closer distanceby changing its shape to reduce the power or focal length. The distanceto which the eye must be focused to create a sharp retinal image is theaccommodative distance. Thus, under normal conditions in the real world,the vergence distance (dv) is equal to the accommodative distance (da).

FIG. 1B shows an example conflict between vergence and accommodationthat can occur with some three-dimensional displays. As shown in FIG.1B, a user is looking at a virtual object 100B displayed on anelectronic screen 104. However, the user's eyes are verged on and gazelines from the user's eyes intersect at virtual object 100B, which is agreater distance from the user's eyes than the electronic screen 104. Asthe virtual object 100B is rendered on the electronic display 104 toappear closer to the user, each eye 102 again rotates inward to stayverged on the virtual object 100B, but the power or focal length of eacheye is not reduced; hence, the user's eyes do not accommodate as in FIG.1A. Thus, instead of reducing power or focal length to accommodate forthe closer vergence distance, each eye 102 maintains accommodation at adistance associated with the electronic display 104. Thus, the vergencedistance (d_(v)) often is not equal to the accommodative distance(d_(a)) for the human eye for objects displayed on 2D electronicdisplays. This discrepancy between vergence distance (d_(v)) andaccommodative distance (d_(a)) is referred to as “vergence-accommodationconflict.” A user who is experiencing only vergence or accommodation butnot both will eventually experience some degree of fatigue and nausea,which is undesirable for virtual reality system creators.

FIG. 2A illustrates a wire diagram of an example head-mounted display(HMD) 200, in accordance with an embodiment of the present disclosure.As shown in FIG. 2A, the HMD 200 may include a front rigid body 205 anda band 710. The front rigid body 205 may include one or more electronicdisplay elements of an electronic display (not shown), an inertialmeasurement unit (IMU) 215, one or more position sensors 220, andlocators 225. In the embodiment shown by FIG. 2A, the position sensors220 may be located within the IMU 215, and neither the IMU 215 nor theposition sensors 220 may be visible to the user. The IMU 215, theposition sensors 220, and the locators 225 may be discussed in detailbelow with regard to FIG. 7. The HMD 200 acts as a virtual reality (VR)device, an augmented reality (AR) device or a mixed reality (MR) device,or some combination thereof. In some embodiments, when the HMD 200 actsas an augmented reality (AR) or a mixed reality (MR) device, portions ofthe HMD 200 and its internal components may be at least partiallytransparent.

FIG. 2B is a cross section 250 of the front rigid body 205 of theembodiment of the HMD 200 shown in FIG. 2A. As shown in FIG. 2B, thefront rigid body 205 may include an electronic display 255, a varifocalblock 260 and an eye tracking system 270. The electronic display 255 maydisplay images (i.e., virtual scenes) to the user. In some embodiments,the electronic display 255 may include a stack of one or more waveguidedisplays 275 including, but not limited to, a stacked waveguide display.The varifocal block 260 may include one or more varifocal structures inoptical series. A varifocal structure is an optical device that isconfigured to dynamically adjust its focus in accordance withinstructions from a varifocal system. The electronic display 255 and thevarifocal block 260 together provide image light to an exit pupil 263.The eye tracking system 270 may include, e.g., one or more sources thatilluminate one or both eyes of the user, and one or more cameras thatcaptures images of one or both eyes of the user. The eye tracking system270 may detect a location or an object in the virtual scene at which theuser's eye 265 is currently looking. The exit pupil 263 may be thelocation of the front rigid body 205 where a user's eye 265 ispositioned. For purposes of illustration, FIG. 2B shows a cross section250 associated with a single eye 265, but another varifocal block, whichis separated from the varifocal block 260, may provide altered imagelight to another eye of the user.

Optical series refers to relative positioning of a plurality of opticalelements, such that light, for each optical element of the plurality ofoptical elements, is transmitted by that optical element before beingtransmitted by another optical element of the plurality of opticalelements. Moreover, ordering of the optical elements does not matter.For example, optical element A placed before optical element B, oroptical element B placed before optical element A, are both in opticalseries. Similar to electric circuitry design, optical series representsoptical elements with their optical properties compounded when placed inseries.

FIG. 3 illustrates an exemplary varifocal structure according to thepresent disclosure. As shown in FIG. 3 and FIG. 2B, the varifocalstructure may include an adaptive lens assembly 300, which may include aplurality of optical lenses 305 arranged in an array or any appropriategeometric formation. The adaptive lens assembly 300 may have a lengthalong the x-axis or horizontal direction and a width along the y-axis orvertical direction.

The plurality of optical lenses 305 may be coupled together to form theadaptive lens assembly 300. For example, the plurality of optical lenses305 may be individual lenses coupled together through mechanical and/orelectrical means, such that the plurality of optical lenses 305 may becontrolled individually and independently. In certain embodiments, theplurality of optical lenses 305 may be integrated together duringfabrication to form a uniform lens. That is, the plurality of opticallenses 305 may be formed at the same time and in the same process as anintegral lens array. Other arrangements may also be used.

The optical lens 305 may include any appropriate lens units, such as aglass lens, a polymer lens, a liquid lens, a liquid crystal (LC) lens,or some combination thereof. The optical lens 305 may adjust anorientation of light emitted from the electronic display 255, such thatthe light emitted from the electronic display 255 appears at particularfocal distances/image planes from the user. In certain embodiments, theoptical lens 305 may be an LC lens, which is capable of adjusting theoptical power sufficiently fast to keep pace with eye accommodation(e.g., accommodation occurs in around 300 ms), such that thevergence-accommodation conflict can be resolved.

In some embodiments, each optical lens 305 may include a plurality oflayers of lens units, each of the layers of lens unit may be referred asa lens layer 310. That is, each optical lens 305 may include at leasttwo lens layers 310, and each layer has a lens unit. The plurality oflens units (i.e., lens layers) may be stacked together to form theoptical lens 305, and the total optical power of the optical lens 305may be a sum of the optical power of the plurality of lens units. Thenthe plurality of optical lenses 305 may be coupled together to form theadaptive lens assembly 300. In some embodiments, the plurality of lensunits (i.e., lens layers) which are stacked together to form the opticallens 305 may be individually controlled. For example, when adjusting thefocal length of the adaptive lens assembly, certain lens units may beactivated while certain lens units may be deactivated, and the activatedlens units may be configured to have same or different optical power. Insome embodiments, the plurality of lens units (i.e., lens layers) whichare stacked together to form the optical lens 305 may be integrallycontrolled. For example, when adjusting the focal length of the adaptivelens assembly, the plurality of lens units may be all activated ordeactivated, and the activated lens units may be configured to have thesame optical power.

For illustrative purposes, FIG. 3 shows each optical lens 305 mayinclude four layers of lens units, and each layer has a lens unit. Thetotal number of lens units or lens layers may be determined based onapplication requirements, such as desired lens resolution, responsetime, and/or optical power, etc. In another perspective, it may also beviewed that the adaptive lens assembly 300 may include a plurality oflens layers 310 (e.g., at least two lens layers) stacked together, andeach lens layer 310 may include a plurality of lens units arranged in anarray. That is, layers of array of the lens units may be stackedtogether to form an array of optical lenses 305, which forms theadaptive lens assembly 300. The total number of lens layers, the totalnumber of lens units in each layer, and the length and width of theadaptive lens assembly 300 may be determined in advance based on variousapplication scenarios.

In some embodiments, the optical lens 305 may be an LC lens 305, andeach lens unit in the optical lens 305 may be also an LC lens. Apredetermined number of lens units (i.e., lens layers) may be stackedtogether to form the LC lens 305. FIG. 4A illustrates an exemplary lensunit 400 in the LC lens 305 consistent with the disclosed embodiments.

As shown in FIG. 4A, the lens unit 400 may include an LC lens with aFresnel structure, i.e., a Fresnel LC lens. The Fresnel LC lens mayinclude any appropriate type of Fresnel structure, such as a Fresnelzone plate lens including areas that have a phase difference of ahalf-wave to adjacent areas, a diffractive Fresnel lens having asegmented parabolic phase profile where the segments are small and canresult in significant diffraction, or a refractive Fresnel lens having asegmented parabolic profile where the segments are large enough so thatdiffraction effects are minimized. Other structures may also be used.

In some embodiments, the lens unit 400 may include a refractive FresnelLC lens having a segmented parabolic profile, where the segments arelarge enough such that the diffraction angle is smaller than the angularresolution of human eyes, i.e., diffraction effects are not observableby human eyes. Such a refractive Fresnel LC lens is referred as asegmented phase profile (SPP) LC lens 400. Referring to FIG. 4A, theFresnel structure of the SPP LC lens 400 is represented by a pluralityof concentric ring-shaped zones 402 of increasing radii, which arereferred as Fresnel segments or Fresnel resets.

For a positive thin lens, optical path difference (OPD) is approximatedwith Maclaurin series to a parabolic profile as shown in Equation (1)

$\begin{matrix}{{{{OPD}(r)} = \frac{r^{2}}{f}},} & (1)\end{matrix}$

where r is the lens radius (i.e., half of the lens aperture) and f isthe focal length. The OPD of an LC lens is proportional to the cellthickness d and the birefringence Δn of the LC material as shown inEquation (2)

OPD=d*Δn,  (2)

The response time τ of an Electrically Controlled Birefringence (ECB) LCcell, which is the time the material requires to recover to its originalstate, is quadratically dependent on cell thickness d (τ∝d²) as shown inEquation (3)

$\begin{matrix}{{\tau = \frac{\gamma \times d^{2}}{K_{11} \times \pi^{2}}},} & (3)\end{matrix}$

where γ and K₁₁ are the rotational viscosity and the splay elasticconstant of the LC material, respectively. Equations (1)-(3) show thereis a tradeoff between the aperture size and response time, and thusdesigning an LC lens with large aperture and reasonable response time isan uphill task. In the disclosed embodiments, though introducing phaseresets in the parabolic phase profile, i.e., using a SPP LC lens, alarge aperture size of the LC lens may be allowed without compromisingthe response time.

FIG. 4B illustrates a desired phase profile for ±0.375 Diopter (D) LClens, where the OPD equals to 35λ. The thickness of the LC cell would beabout 70 μm for LC materials with a birefringence value of 0.27. Todecrease the effective thickness of the LC cell, resets or segments maybe introduced into the lens phase profile. FIG. 4C illustrates 2D phasemap of the SPP LC lens 400 with 5 resets, the thickness of the LC cellwould be reduced up to 5 times and, accordingly, the response time wouldbe improved by a factor of 25. That is, through introducing the segmentsin the lens phase profile, the optical power of the SPP LC lens 400 maybe adjusted sufficiently fast to keep pace with eye accommodation (e.g.,accommodation occurs in around 300 ms), such that thevergence-accommodation conflict would be resolved. The number of theresets may be determined based on specific configurations of the Fresnelstructure and the SPP LC lens 400 requirements, such as the desiredoptical power, lens aperture, switching time, image quality of the LClens. A large number of phase steps within one wavelength of OPD (i.e.,per wavelength) may be desired for accurate representation of phaseprofile. Meanwhile, to configure the SPP LC lens with negligiblediffraction angle for near eye applications, the minimum width of theFresnel segments (i.e., the minimum Fresnel segment width) of the SPP LClens 400, for green wavelength of 543.5 nm, is desired to be larger than1.03 mm.

FIG. 4D illustrates a partial cross-sectional view of the SPP LC lens400. As shown in FIG. 4D, the SPP LC lens 400 may include a plurality offirst electrodes 412, one or more second electrode 410, a liquid crystal(LC) layer 414, and substrates 416. The substrates 416 may besubstantially transparent in the visible band (˜380 nm to 750 nm). Incertain embodiments, the substrates 416 may also be transparent in someor all of the infrared (IR) band (˜750 nm to 1 mm). The substrate layersmay be composed of, e.g., SiO₂, plastic, sapphire, etc. The firstelectrodes 412 and second electrodes 410 may be transparent electrodes(e.g., indium tin oxide electrodes) disposed on the substrates 416 togenerate electric fields, which reorients the LC molecules in the LClayer 414 to form a lens having a desired phase profile.

In some embodiments, the first electrodes 412 may include discretering-shaped electrodes corresponding to the Fresnel structures in theSPP LC lens 400, and the ring-shaped electrodes may be concentric withidentical area. With this electrode geometry, when the phase differencebetween adjacent first electrodes 412 is the same, a parabolic phaseprofile may be obtained. If the phase is proportional to the appliedvoltage, a linear change in the voltage across the first electrodes 412(same difference in voltage between any two first electrodes 412) mayyield a desired parabolic phase profile.

Further, the gaps between the first electrodes 412 can cause scatteringand thus image degradation. To address that image degradation, as shownin FIG. 4E, a plurality of floating electrodes 418 may be disposed onthe substrate 416 provided with the first electrodes 512. The floatingelectrodes 418 may include discrete and concentric ring electrodes whichare not driven by ohmic connection but are capacitively coupled to thefirst electrodes 412. The floating electrodes 418 may be configured tocover half of the area of each of neighboring first electrodes 412. Aninsulating layer 420 may be disposed between the floating electrodes 418and the first electrodes 412.

To further improve the response time of the SPP LC lens, multiple SPP LClens (i.e., multiple lens layers) may be optically coupled to form astack of SPP LC lens, i.e., an SPP LC lens stack, such that given a sametunable optical power range, the thickness of each SPP LC lens may bereduced and, accordingly, the response of each SPP LC lens may bereduced. For illustrative purposes, FIG. 5 shows a pair of SPP LC lensesmay be optically coupled to form an SPP LC lens stack. Provided thateach SPP LC lens has 5 resets in the phase profile, considering theeffect of the pair of lenses and the Fresnel resets, the thickness ofthe LC cell may be reduced up to 10 times (5 resets×2) and, accordingly,the response speed may be improved by a factor of 100.

Further, the two SPP LC lenses may have opposite rubbing directions oralignment directions on the corresponding LC surfaces of the two SPP LClens, so as to improve the viewing angle. That is, for viewing angleimprovement, two of SPP LC lenses with the same configuration butopposite rubbing directions may be optically coupled. The polarizationinsensitivity is very important for AR configuration. Most LC materialsare birefringent and, thus, are polarization sensitive. When the lightpropagating in a direction parallel to the LC director is incident ontothe LC cell, the light will experience ordinary refractive index nO ofthe LC material for any polarization states. However, when the lightpropagating in a direction perpendicular to the LC director is incidentonto the LC cell, the light will experience refractive index between theordinary refractive index nO and extraordinary refractive index ne ofthe LC material, depending on the polarization state of light.Cholesteric LC materials can be made polarization insensitive asdiscussed by Clarke et al. in Electro-active lens U.S. Pat. No.7,728,949B2. In this case the pitch of cholesteric LCs can be made inthe range of the wavelength of incident light and, therefore, when novoltage is applied to the LC cell, the light will experience an averagerefractive index (n_(o)+n_(e)/2) for any polarization states of light.For nematic LCs, the SPP LC lenses may be configured to be polarizationinsensitive by optically coupling cells of orthogonal polarization, inwhich each cell may focus one polarization state of light, for example,one cell focuses s polarization and the other focuses p polarization.

Returning to FIG. 3, to enable the LC lens 305 to adjust the opticalpower sufficiently fast to keep pace with eye accommodation (e.g.,accommodation occurs in around 300 ms), each LC lens 305 may be formedby lens layers of SPP LC lenses, and each SPP LC lens may be configuredto have a reduced aperture size. That is, in addition to use more lenslayers of the SPP LC lenses, each layer may include an array of SPP LClenses with a reduced aperture size, and the array of SPP LC lenses maybe fabricated together at the same time as a single lens layer, e.g., ona single wafer.

Below various designs of varifocal structures are discussed. FIGS. 6A-6Cillustrate exemplary configurations of the SPP LC lens array in a singlelens layer. As shown in FIG. 6A, the array of SPP LC lenses may includeidentical individual SPP LC lenses configured in an overlapped format.In this configuration, the individual SPP LC lenses may be overlapped inboth the horizontal direction and the vertical direction. That is, theelectrode patterns of the neighboring SPP LC lenses may have overlap. Inone embodiment, the array of SPP LC lenses may have a length of 60 mmand a width of 50 mm, and the aperture size of a single SPP LC lens maybe 20 mm. In this configuration, four SPP LC lenses may be arranged inthe horizontal direction and three SPP LC lenses may be arranged in thevertical direction.

As shown in FIG. 6B, the array of SPP LC lenses may include identicalindividual SPP LC lenses configured in a non-overlapped format. In thisconfiguration, the individual SPP LC lenses 305 may be arranged in boththe horizontal direction and the vertical direction. That is, theelectrode patterns of the neighboring SPP LC lenses 305 may have nooverlap. However, in areas between neighboring SPP LC lenses 305, e.g.,area 1 in FIG. 6B, the electric field of each area may be adjusted suchthat the phase profile of individual SPP LC lens 305 may be preserved.In one embodiment, the array of SPP LC lenses may have a length of 60 mmand a width of 60 mm, and the size of a single SPP LC lens may be 20 mm.In this configuration, three SPP LC lenses may be arranged in thehorizontal direction and three SPP LC lenses may be arranged in thevertical direction.

As shown in FIG. 6C, the array of SPP LC lenses may include differentlevels of SPP LC lenses, and each level may include identical individualSPP LC lenses configured in a non-overlapped format. In thisconfiguration, the individual SPP LC lenses may be arranged in both thehorizontal direction and the vertical direction without overlap, but alower level of SPP LC lenses may be arranged in areas between a currentlevel of SPP LC lenses until the SPP LC lenses have a diameter smallerthan the eye resolution (about 1 arc min). That is, the electrodepatterns of the SPP LC lenses may have no overlap. However, thedifferent levels of SPP LC lenses may be configured to preserve a moreuniform parabolic phase profile of individual SPP LC lens. For example,a second-level of SPP LC lenses may be disposed in areas betweenneighboring SPP LC lenses, and a third-level of SPP LC lenses may bedisposed in areas between neighboring second-level SPP LC lenses. In oneembodiment, the array of SPP LC lenses may have a length of 60 mm and awidth of 60 mm, and the size of a single top level SPP LC lens may be 20mm, three levels of SPP LC lenses may be included.

For illustrative purposes, FIGS. 6A-6C merely illustrate exemplaryconfigurations of the SPP LC lens array in a single lens layer.Referring to FIG. 3, the adaptive lens assembly 300 may include aplurality of lens layers 310 (e.g., at least two lens layers) stackedtogether, and each lens layer 310 may include a plurality of lens unitsarranged in an array, in which each lens layer may be any one of thesingle lens layers shown in FIGS. 6A-6C. The formed adaptive lensassembly 300 may replace a 50-mm-aperture SPP LC lens, for example, theformed adaptive lens assembly 300 in which each lens layer includes 4×320-mm-aperture SPP LC lens array may replace a 50-mm-aperture SPP LClens, such that the tunable optical power range, response time, and/orresolution of the lens of the individual SPP LC lenses may be configuredin desired ranges and the overall lens properties of the adaptive lensassembly 300 may also be in a desired range for VR, AR, MR applications,or some combination thereof.

Referring to FIG. 2B, FIG. 3 and FIGS. 6A-6C, in an operation of theHMD, the eye tracking system 270 may detect an eye position for each eyeof the user. For example, the eye tracking system 270 may determine alocation or an object in the displayed virtual scene/image at which theuser's eye 265 is currently looking. Then the SPP LC lens(es) 305corresponding to the location or the object in the virtual scene atwhich the user's eye 265 is currently looking may be activated, and theoptical power of the activated SPP LC lens(es) may be adjusted toaddress the vergence-accommodation conflict. However, because only oneor a few SPP LC lenses in the lens array may be activated at one time,the center of the activated SPP LC lens(es) 305, which is referred as alocal lens center, may be different from the center of the adaptive lensassembly 300. That is, a shift may exist between the local lens centerand the center of the adaptive lens assembly 300, i.e., a lens centershift. As a result, the user's eye 265 may observe an angular shift ofthe displayed image caused by the lens center shift. To compensate forthe angular shift caused by the lens center shift, the electronicdisplay 255 may shift the displayed image along with the lens centershift, such that the user's eye 265 does not feel any shift on thedisplayed image. That is, an image shift may be introduced to compensatefor the angular shift caused by the lens center shift, where the imageshift is a shift between the same point in the displayed image beforeshift and after shift. For example, the image shift is a shift betweenthe center of the displayed image before shift and after shift.

FIG. 6D illustrates exemplary lens center shift and image shiftaccording to the present disclosure. As shown in FIG. 6D, the varifocalblock may include a main lens 245 in addition to the adaptive lensassembly 300. After the SPP LC lens(es) corresponding to the location orthe object in the displayed image at which the user's eye 265 iscurrently looking is activated, the lens center shift is d. To displayan on-axis image point C to be observed by the eye 265, thecorresponding image point A (shifted image point A) displayed on theelectronic display 255 is shifted by 4, i.e., the image shift is 4. Theimage point B is a virtual point of the shifted image point A formed bythe main lens 245, and the image point C is a virtual point of theshifted image point A to be observed by the eye 265. t is a distancebetween the main lens 245 and the electronic display 255, and the valueoft may depend on design requirements of the size and field of view(FOV) of the electronic display 255. The main lens 245 may be configuredto have a focal length F greater than t (i.e., F>t), such that thevirtual image formed by the main lens is at some finite distance. Inthis way, the adaptive lens assembly 300 may be configured to providenegative or positive optical power depending on the displayed virtualimage distance. For the shifted image point A, t and s are an objectdistance and an image distance of the image point A with respect to themain lens 245. The image shift on the electronic display 255 iscalculated as 4=d*t/s. For example, when a displayed image is desired tobe observed at 0.5 m to infinity (i.e., 2D), given s=1 m, then theoptical power range of the adaptive lens assembly 300 would be −1D to+1D. When t˜30 mm, the image shift Δ=˜0.03d. That is, when the lenscenter shift is 20 mm, the image shift may be 0.6 mm.

Thus, based on the above approaches, the response time, the resolution,the tunable optical power range, and/or the image quality of theadaptive lens assembly 300 may be in a desired range for VR, AR, and MRapplications, or some combination thereof. For AR or MR applications,another adaptive lens assembly may be introduced to compensate thedistortion of the real-world images caused by the adaptive lens assembly300. The another adaptive lens assembly may provide optical power, whichis opposite to but having a same absolute value as the optical powerprovided by the adaptive lens assembly 300, such that the real-worldobjects viewed through the HMD may stay unaltered.

FIG. 7 illustrates an exemplary varifocal system 700 including certainaspects of disclosed embodiments. The varifocal system 700 may be usedfor a VR system, an AR system, a MR system, or some combination thereof.As shown in FIG. 7, the varifocal system 700 may include an imagingdevice 710, a console 720, an input/output interface 715, and ahead-mounted display (HMD) 200. Certain device(s) may be omitted, andother devices or components may also be included. Although FIG. 7 showsa single HMD 705, a single imaging device 710, and a single input/outputinterface 715, any number of these components may be included in thevarifocal system 700. The HMD 705 may act as a VR, AR, and/or a MR HMD.

The HMD 705 may present content to a user. In some embodiments, the HMD705 may be an embodiment of the HMD 200 described above with referenceto FIGS. 2A and 2B. Example content includes images, video, audio, orsome combination thereof. Audio content may be presented via a separatedevice (e.g., speakers and/or headphones) external to the HMD 705 thatreceives audio information from the HMD 705, the console 820, or both.The HMD 705 may include an electronic display 255 (described above withreference to FIG. 2B), a varifocal block 260 (described above withreference to FIG. 2B), an eye tracking system 270, a vergence processingmodule 730, one or more locators 225, an internal measurement unit (IMU)215, head tracking sensors 735, and a scene rendering module 740.

The eye tracking system 270 may track an eye position and eye movementof a user of the HMD 705. A camera or other optical sensor (that is partthe eye tracking system 270) inside the HMD 705 may capture imageinformation of a user's eyes, and the eye tracking system 270 may usethe captured information to determine interpupillary distance,interocular distance, a three dimensional (3D) position of each eyerelative to the HMD 705 (e.g., for distortion adjustment purposes),including a magnitude of torsion and rotation (i.e., roll, pitch, andyaw) and gaze directions for each eye.

In some embodiments, infrared light may be emitted within the HMD 705and reflected from each eye. The reflected light may be received ordetected by the camera and analyzed to extract eye rotation from changesin the infrared light reflected by each eye. Many methods for trackingthe eyes of a user may be used by eye tracking system 270. Accordingly,the eye tracking system 270 may track up to six degrees of freedom ofeach eye (i.e., 3D position, roll, pitch, and yaw), and at least asubset of the tracked quantities may be combined from two eyes of a userto estimate a gaze point (i.e., a 3D location or position in the virtualscene where the user is looking). For example, the eye tracking system270 may integrate information from past measurements, measurementsidentifying a position of a user's head, and 3D information describing ascene presented by the electronic display 255. Thus, information for theposition and orientation of the user's eyes is used to determine thegaze point in a virtual scene presented by the HMD 705 where the user iscurrently looking.

The varifocal block 260 may adjust its focal length (i.e., opticalpower) by adjusting a focal length of one or more varifocal structures.As noted above with reference to FIGS. 6A-6C, based on the eye trackinginformation, the varifocal block 260 may activate one or more LC lensescorresponding to the eye position for each eye of the user, and adjustits focal length by adjusting the voltages applied to the electrodes ofthe one or more activated LC lenses. The varifocal block 260 may adjustits focal length responsive to instructions from the console 720. Notethat a varifocal tuning speed of a varifocal structure is limited by atuning speed of the LC lenses. The varifocal block 260 may deactivateother LC lenses which are not corresponding to the eye position for eacheye of the user, thereby reducing the power consumption of the varifocalblock 260. In addition, the varifocal block 260 may determine a shiftbetween the center of the activated LC lens(es) and the center of theadaptive lens assembly, i.e., a lens center shift.

The vergence processing module 730 may determine a vergence distance ofa user's gaze based on the gaze point or an estimated intersection ofthe gaze lines determined by the eye tracking system 270. Vergence isthe simultaneous movement or rotation of both eyes in oppositedirections to maintain single binocular vision, which is naturally andautomatically performed by the human eye. Thus, a location where auser's eyes are verged is where the user is currently looking and isalso typically the location where the user's eyes are currently focused.For example, the vergence processing module 730 may triangulate the gazelines to estimate a distance or depth from the user associated withintersection of the gaze lines. Then the depth associated withintersection of the gaze lines may be used as an approximation for theaccommodation distance, which identifies a distance from the user wherethe user's eyes are directed. Thus, the vergence distance may allow thedetermination of a location where the user's eyes should be focused.

The locators 225 may be objects located in specific positions on the HMD705 relative to one another and relative to a specific reference pointon the HMD 705. A locator 225 may be a light emitting diode (LED), acorner cube reflector, a reflective marker, a type of light source thatcontrasts with an environment in which the HMD 705 operates, or somecombination thereof.

The IMU 215 may be an electronic device that generates fast calibrationdata based on measurement signals received from one or more of the headtracking sensors 735, which generate one or more measurement signals inresponse to motion of HMD 705. Examples of the head tracking sensors 735include accelerometers, gyroscopes, magnetometers, other sensorssuitable for detecting motion, correcting error associated with the IMU215, or some combination thereof.

Based on the measurement signals from the head tracking sensors 735, theIMU 215 may generate fast calibration data indicating an estimatedposition of the HMD 705 relative to an initial position of the HMD 705.For example, the head tracking sensors 735 may include multipleaccelerometers to measure translational motion (forward/back, up/down,left/right) and multiple gyroscopes to measure rotational motion (e.g.,pitch, yaw, and roll). The IMU 215 may, for example, rapidly sample themeasurement signals and calculate the estimated position of the HMD 705from the sampled data. Alternatively, the IMU 215 may provide thesampled measurement signals to the console 720, which determines thefast calibration data.

The IMU 215 may additionally receive one or more calibration parametersfrom the console 720. As further discussed below, the one or morecalibration parameters may be used to maintain tracking of the HMD 705.Based on a received calibration parameter, the IMU 215 may adjust one ormore of the IMU parameters (e.g., sample rate). In some embodiments,certain calibration parameters may cause the IMU 215 to update aninitial position of the reference point to correspond to a nextcalibrated position of the reference point. Updating the initialposition of the reference point as the next calibrated position of thereference point may help to reduce accumulated error associated withdetermining the estimated position. The accumulated error, also referredto as drift error, may cause the estimated position of the referencepoint to “drift” away from the actual position of the reference pointover time.

The scene rendering module 740 may receive contents for the virtualscene from a VR engine 745, and provide display content for display onthe electronic display 255. The scene rendering module 740 may include ahardware central processing unit (CPU), graphic processing unit (GPU),and/or a microcontroller. Additionally, the scene rendering module 740may adjust the content based on information from the eye tracking system270, the vergence processing module 730, the IMU 215, and the headtracking sensors 735. The scene rendering module 740 may determine aportion of the content to be displayed on the electronic display 255,based on one or more of the eye tracking system 270, the tracking module755, the head tracking sensors 735, or the IMU 215. For example, thescene rendering module 740 may determine a virtual scene to be displayedto the viewer's eyes, or any part of the virtual scene. The scenerendering module 740 may also dynamically adjust the displayed contentbased on the real-time configuration of the varifocal block 260. Inaddition, based on the information of the determined lens center shiftprovided by the varifocal block 260, the scene rendering module 740 maydetermine a shift of the virtual scene to be displayed on the electronicdisplay 255.

The imaging device 710 may provide a monitoring function for the HMD 705and may generate slow calibration data in accordance with calibrationparameters received from the console 720. Slow calibration data mayinclude one or more images showing observed positions of the locators225 that are detectable by imaging device 710. The imaging device 710may include one or more cameras, one or more video cameras, otherdevices capable of capturing images including one or more locators 225,or some combination thereof. Slow calibration data may be communicatedfrom the imaging device 710 to the console 720, and the imaging device710 may receive one or more calibration parameters from the console 720to adjust one or more imaging parameters (e.g., focal length, focus,frame rate, ISO, sensor temperature, shutter speed, aperture, etc.).

The input/output interface 715 may be a device that allows a user tosend action requests to the console 720. An action request may be arequest to perform a particular action. For example, an action requestmay be to start or end an application or to perform a particular actionwithin the application. The input/output interface 715 may include oneor more input devices such as a keyboard, a mouse, a game controller, orany other suitable device. An action request received by theinput/output interface 715 may be communicated to the console 720, whichperforms an action corresponding to the action request. In someembodiments, the input/output interface 715 may provide haptic feedbackto the user in accordance with instructions received from the console720. For example, haptic feedback may be provided by the input/outputinterface 715 when an action request is received, or the console 720 maycommunicate instructions to the input/output interface 715 causing theinput/output interface 715 to generate haptic feedback when the console720 performs an action.

The console 720 may provide content to the HMD 705 for presentation tothe user in accordance with information received from the imaging device710, the HMD 705, or the input/output interface 715. In one embodiment,as shown in FIG. 7, the console 720 may include an application store750, a tracking module 755, and the VR engine 745, etc.

The application store 750 may store one or more applications forexecution by the console 720. An application may be a group ofinstructions, that when executed by a processor, generates content forpresentation to the user. Content generated by an application may be inresponse to inputs received from the user via movement of the HMD 705 orthe input/output interface 715. Examples of applications include gamingapplications, conferencing applications, video playback application, orother suitable applications.

The tracking module 755 may calibrate the varifocal system 700 using oneor more calibration parameters and may adjust one or more calibrationparameters to reduce error in determining position of the HMD 705. Forexample, the tracking module 755 may adjust the focus of the imagingdevice 710 to obtain a more accurate position for observed locators 225on the HMD 705. Moreover, calibration performed by the tracking module755 may also account for information received from the IMU 215.Additionally, when tracking of the HMD 705 is lost (e.g., imaging device710 loses line of sight of at least a threshold number of locators 225),the tracking module 755 may re-calibrate some or all of the varifocalsystem 700 components.

Additionally, the tracking module 755 may track the movement of the HMD705 using slow calibration information from the imaging device 710, anddetermine positions of a reference point on the HMD 705 using observedlocators from the slow calibration information and a model of the HMD705. The tracking module 755 may also determine positions of thereference point on the HMD 705 using position information from the fastcalibration information from the IMU 215 on the HMD 705. Additionally,the tracking module 755 may use portions of the fast calibrationinformation, the slow calibration information, or some combinationthereof, to predict a future location of the HMD 705, which is providedto the VR engine 745.

The VR engine 745 may execute applications within the varifocal system700 and receive position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof forthe HMD 705 from the tracking module 755. Based on the receivedinformation, the VR engine 745 may determine content to provide to theHMD 705 for presentation to the user, such as a virtual scene, one ormore virtual objects to overlay onto a real-world scene, etc. In someembodiments, the VR engine 845 may maintain focal capability informationof the varifocal block 260. Focal capability information is informationthat describes what focal distances are available to the varifocal block260. Focal capability information may include, e.g., a range of focusthat the varifocal block 260 is able to accommodate (e.g., 0 to 4diopters), combinations of settings for each activated LC lens that mapto particular focal planes; or some combination thereof.

The VR engine 745 may provide information to the varifocal block 260,such as the accommodation and/or convergence parameters including whatfocal distances are available to the varifocal block 260. The VR engine745 may generate instructions for the varifocal block 260, theinstructions causing the varifocal block 260 to adjust its focaldistance to a particular location. The VR engine 745 may generate theinstructions based on focal capability information and, e.g.,information from the vergence processing module 730, the IMU 215, andthe head tracking sensors 735, and provide the instructions to thevarifocal block 260 to configure and/or adjust the adaptive assembly260. The VR engine 745 may use the information from the vergenceprocessing module 730, the IMU 215, and the head tracking sensors 735,or some combination thereof, to select a focal plane to present contentto the user. Additionally, the VR engine 745 may perform an actionwithin an application executing on the console 720 in response to anaction request received from the input/output interface 715, and providefeedback to the user that the action was performed. The providedfeedback may be visual or audible feedback via the HMD 705 or hapticfeedback via the input/output interface 715.

FIG. 8 is a process 800 for mitigating vergence-accommodation conflictby adjusting the focal length of the HMD 705 according to the presentdisclosure. The process 900 may be performed by the varifocal system 700in some embodiments. Alternatively, other components may perform some orall of the steps of the process 800. For example, in some embodiments,an HMD 705 and/or a console (e.g., console 720) may perform some of thesteps of the process 800. Additionally, the process 800 may includedifferent or additional steps than those described in conjunction withFIG. 8 in some embodiments or perform steps in different orders than theorder described in conjunction with FIG. 8. Additionally, the process800 may include different or additional steps than those described inconjunction with FIG. 8 in some embodiments or perform steps indifferent orders than the order described in conjunction with FIG. 8.

Referring to FIG. 7 and FIG. 8, the varifocal system 700 may determine aposition, an orientation, and/or a movement of HMD 705 (Step 810). Theposition may be determined by a combination of the locators 225, the IMU215, the head tracking sensors 735, the imagining device 710, and thetracking module 755, etc., as described above in conjunction with FIG.7.

The varifocal system 700 may determine a portion of a virtual scenebased on the determined position and orientation of the HMD 705 (Step820). The varifocal system 700 may map a virtual scene presented by theHMD 705 to various positions and orientations of the HMD 705. Thus, aportion of the virtual scene currently viewed by the user may bedetermined based on the position, orientation, and movement of the HMD705.

The varifocal system 700 may display the determined portion of thevirtual scene being on an electronic display (e.g., the electronicdisplay 255) of the HMD 705 (Step 830). In some embodiments, the portionmay be displayed with a distortion correction to correct for opticalerror that may be caused by the image light passing through thevarifocal block 260.

The varifocal system 700 may determine an eye position for each eye ofthe user using an eye tracking system (Step 840). The varifocal system700 may determine a location or an object within the determined portionat which the user is looking to adjust focus for that location or objectaccordingly. To determine the location or object within the determinedportion of the virtual scene at which the user is looking, the HMD 705may track the position and location of the user's eyes using imageinformation from (e.g., the eye tracking system 270) of the HMD 705. Forexample, the HMD 705 may track at least a subset of a 3D position, roll,pitch, and yaw of each eye, and use these quantities to estimate a 3Dgaze point of each eye.

Further, based on the eye tracking information, the varifocal system 700may determine a desired optical power of the HMD 705 based on a vergencedistance (Step 850). In some embodiment, the varifocal system 700 maydetermine the vergence distance based on an estimated intersection ofgaze lines. In some embodiments, information from past eye positions,information describing a position of the user's head, and informationdescribing a scene presented to the user may be used to estimate the 3Dgaze point of an eye. The optical power required may then be calculatedbased on the vergence distance of the virtual scene and otherinformation.

Based on the determined optical power of the HMD 705 and the eyetracking information, the varifocal system 700 may determineconfiguration parameters for the LC lenses in the varifocal block 260(Step 860). In particular, based on the eye tracking information, thevarifocal system 700 may activate one or more LC lenses corresponding tothe eye position for each eye of the user and, meanwhile, based on thedetermined optical power, the varifocal system 700 may determine desiredvoltages to be applied to the electrodes of the one or more activated LClens.

For example, as the varifocal block 260 includes an array of LC lenseseach having a predetermined number of layers of SPP LC lens units, thevarifocal system 700 may determine which LC lens or lenses in the arrayshould be used. In one embodiment, based on the 3D gaze point of theeye, the varifocal system 700 may determine an LC lens that intersectswith the gaze line of the eye, and may select that LC lens to beactivated. In some embodiments, when the gaze line of the eye falls inan area between neighboring LC lenses, the varifocal system 700 may alsoselect one or more neighboring LC lenses to be activated or may selectan SPP LC lens closest to the gaze line to be activated.

The varifocal system 700 may also determine an overall desired opticalpower value for the selected LC lens or lenses. As the selected LC lenshas a plurality of layers of SPP LC lenses, and each SPP LC lens mayhave a specific optical power range, certain or all layers of SPP LClenses may be selected based on the specific optical power range and theoverall desired optical power. The varifocal system 700 may select thelens layers arranged in sequence to satisfy the overall optical power,or may randomly select the lens layers to satisfy the overall opticalpower. In one embodiment, the varifocal system 700 may select the layersof SPP LC lenses in pairs as shown in FIG. 7 to improve the viewingangle. In addition, the varifocal system 700 may also select the layersof SPP LC lenses so as to satisfy the polarization insensitive state ofthe selected layers of SPP LC lenses. Other selecting criteria may alsobe used.

After the configurations of the SPP LC lenses of the varifocal block 260are determined, the varifocal system 700 may adjust the SPP LC lensesbased on the configurations (Step 870). For example, for each selectedSPP LC lens to be activated in each selected layer of SPP LC lenses tobe activated, the varifocal system 700 may apply the determined voltageson the electrodes of the selected SPP LC lens(es) in each selected layerto activate the selected SPP LC lenses. The varifocal system 700 maykeep other unselected SPP LC lenses inactivated. Thus, accommodation maybe provided for the determined vergence distance corresponding to whereor what in the displayed portion of the virtual scene the user iscurrently looking.

Further, based on the position of the activated LC lens(es), thevarifocal system 700 may adjust the displayed virtual scene to correctthe lens center shift (S880). In particular, because only one or a fewLC lenses in the lens array in the adaptive lens assembly may beactivated at one time, the center of the activated LC lens or the lensesmay be different from the center of the entire adaptive assembly. Whenit is determined that the center of the activated SPP LC lens or lensesis different from the center of the adaptive lens assembly, a shiftbetween the two centers, i.e., a lens center shift, may be calculated.Based on the calculated lens center shift, the image displayed on theelectronic display may be shifted accordingly to compensate for the lenscenter shift, such that the viewer does not feel any shift on thedisplayed image.

The foregoing description of the embodiments of the disclosure have beenpresented for the purpose of illustration. It is not intended to beexhaustive or to limit the disclosure to the precise forms disclosed.Persons skilled in the relevant art can appreciate that manymodifications and variations are possible in light of the abovedisclosure.

Some portions of this description describe the embodiments of thedisclosure in terms of algorithms and symbolic representations ofoperations on information. These algorithmic descriptions andrepresentations are commonly used by those skilled in the dataprocessing arts to convey the substance of their work effectively toothers skilled in the art. These operations, while describedfunctionally, computationally, or logically, are understood to beimplemented by computer programs or equivalent electrical circuits,microcode, or the like. Furthermore, it has also proven convenient attimes, to refer to these arrangements of operations as modules, withoutloss of generality. The described operations and their associatedmodules may be embodied in software, firmware, hardware, or anycombinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by a computer processor for performing any or allof the steps, operations, or processes described.

Embodiments of the disclosure may also relate to an apparatus forperforming the operations herein. This apparatus may be speciallyconstructed for the required purposes, and/or it may comprise ageneral-purpose computing device selectively activated or reconfiguredby a computer program stored in the computer. Such a computer programmay be stored in a non-transitory, tangible computer readable storagemedium, or any type of media suitable for storing electronicinstructions, which may be coupled to a computer system bus.Furthermore, any computing systems referred to in the specification mayinclude a single processor or may be architectures employing multipleprocessor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that isproduced by a computing process described herein. Such a product maycomprise information resulting from a computing process, where theinformation is stored on a non-transitory, tangible computer readablestorage medium and may include any embodiment of a computer programproduct or other data combination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the disclosure be limited notby this detailed description, but rather by any claims that issue on anapplication based hereon. Accordingly, the disclosure of the embodimentsis intended to be illustrative, but not limiting, of the scope of thedisclosure, which is set forth in the following claims.

What is claimed is:
 1. A system, comprising: a display configured tooutput a virtual image; a lens assembly optically coupled to the displayand including a plurality of optical lenses; and a controller configuredto: selectively activate one or more of the plurality of optical lenses,determine a lens center shift between a center of the selectivelyactivated one or more of the plurality of optical lenses and a center ofthe lens assembly, and determine an image shift based on the lens centershift for shifting the virtual image output from the display.